Rethinking the Expressive Power of GNNs via Graph Biconnectivity

Designing expressive Graph Neural Networks (GNNs) is a central topic in learning graph-structured data. While numerous approaches have been proposed to improve GNNs in terms of the Weisfeiler-Lehman (WL) test, generally there is still a lack of deep understanding of what additional power they can systematically and provably gain. In this paper, we take a fundamentally different perspective to study the expressive power of GNNs beyond the WL test. Specifically, we introduce a novel class of expressivity metrics via graph biconnectivity and highlight their importance in both theory and practice. As biconnectivity can be easily calculated using simple algorithms that have linear computational costs, it is natural to expect that popular GNNs can learn it easily as well. However, after a thorough review of prior GNN architectures, we surprisingly find that most of them are not expressive for any of these metrics. The only exception is the ESAN framework (Bevilacqua et al., 2022), for which we give a theoretical justification of its power. We proceed to introduce a principled and more efficient approach, called the Generalized Distance Weisfeiler-Lehman (GD-WL), which is provably expressive for all biconnectivity metrics. Practically, we show GD-WL can be implemented by a Transformer-like architecture that preserves expressiveness and enjoys full parallelizability. A set of experiments on both synthetic and real datasets demonstrates that our approach can consistently outperform prior GNN architectures.Comment: ICLR 2023 notable top-5%; 58 pages, 11 figure

Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective

Designing neural networks with bounded Lipschitz constant is a promising way to obtain certifiably robust classifiers against adversarial examples. However, the relevant progress for the important $\ell_\infty$ perturbation setting is rather limited, and a principled understanding of how to design expressive $\ell_\infty$ Lipschitz networks is still lacking. In this paper, we bridge the gap by studying certified $\ell_\infty$ robustness from a novel perspective of representing Boolean functions. We derive two fundamental impossibility results that hold for any standard Lipschitz network: one for robust classification on finite datasets, and the other for Lipschitz function approximation. These results identify that networks built upon norm-bounded affine layers and Lipschitz activations intrinsically lose expressive power even in the two-dimensional case, and shed light on how recently proposed Lipschitz networks (e.g., GroupSort and $\ell_\infty$-distance nets) bypass these impossibilities by leveraging order statistic functions. Finally, based on these insights, we develop a unified Lipschitz network that generalizes prior works, and design a practical version that can be efficiently trained (making certified robust training free). Extensive experiments show that our approach is scalable, efficient, and consistently yields better certified robustness across multiple datasets and perturbation radii than prior Lipschitz networks. Our code is available at https://github.com/zbh2047/SortNet.Comment: 37 pages; to appear in NeurIPS 2022 (Oral

Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective

Recent studies have discovered that Chain-of-Thought prompting (CoT) can dramatically improve the performance of Large Language Models (LLMs), particularly when dealing with complex tasks involving mathematics or reasoning. Despite the enormous empirical success, the underlying mechanisms behind CoT and how it unlocks the potential of LLMs remain elusive. In this paper, we take a first step towards theoretically answering these questions. Specifically, we examine the capacity of LLMs with CoT in solving fundamental mathematical and decision-making problems. We start by giving an impossibility result showing that any bounded-depth Transformer cannot directly output correct answers for basic arithmetic/equation tasks unless the model size grows super-polynomially with respect to the input length. In contrast, we then prove by construction that autoregressive Transformers of a constant size suffice to solve both tasks by generating CoT derivations using a commonly-used math language format. Moreover, we show LLMs with CoT are capable of solving a general class of decision-making problems known as Dynamic Programming, thus justifying its power in tackling complex real-world tasks. Finally, extensive experiments on four tasks show that, while Transformers always fail to predict the answers directly, they can consistently learn to generate correct solutions step-by-step given sufficient CoT demonstrations.Comment: 33 page

A Complete Expressiveness Hierarchy for Subgraph GNNs via Subgraph Weisfeiler-Lehman Tests

Recently, subgraph GNNs have emerged as an important direction for developing expressive graph neural networks (GNNs). While numerous architectures have been proposed, so far there is still a limited understanding of how various design paradigms differ in terms of expressive power, nor is it clear what design principle achieves maximal expressiveness with minimal architectural complexity. Targeting these fundamental questions, this paper conducts a systematic study of general node-based subgraph GNNs through the lens of Subgraph Weisfeiler-Lehman Tests (SWL). Our central result is to build a complete hierarchy of SWL with strictly growing expressivity. Concretely, we prove that any node-based subgraph GNN falls into one of the six SWL equivalence classes, among which $\mathsf{SSWL}$ achieves the maximal expressive power. We also study how these equivalence classes differ in terms of their practical expressiveness such as encoding graph distance and biconnectivity. In addition, we give a tight expressivity upper bound of all SWL algorithms by establishing a close relation with localized versions of Folklore WL tests (FWL). Overall, our results provide insights into the power of existing subgraph GNNs, guide the design of new architectures, and point out their limitations by revealing an inherent gap with the 2-FWL test. Finally, experiments on the ZINC benchmark demonstrate that $\mathsf{SSWL}$-inspired subgraph GNNs can significantly outperform prior architectures despite great simplicity.Comment: 74 pages, 13 figure

Erratum:Surface anisotropy induced spin wave nonreciprocity in epitaxial La0.33Sr0.67MnO3film on SrTiO3substrate (Appl. Phys. Lett. (2020) 117 (232402)

In the original published article,1the concentrations of La and Sr are reversed. The correct concentration should be La0.67Sr0.33MnO3(which is in the ferromagnetic phase) rather than La0.33Sr0.67MnO3(which is in the antiferromagnetic phase) in the original published version. They were typos of the element concentrations. This misprint does not change the identification or conclusion presented in the original published paper

An Traffic Control Method Based on the Real-Time Detector Delay

This paper puts forward a new single intersection control optimization method based on real-time delays by using the new technology of traffic detection. With this method the delay data (including parking delay, acceleration and deceleration delays) of vehicles and speed of the vehicles at the stop bar in every cycle can be detected real-timely. Thus, we can change the cycle length, total split, yellow time and all red time according to the analysis of the reasons of delay changes so as to optimize the traffic signal timing. Compared with traditional methods, this method can effectively deal with the situation that the traffic control scheme does not conform to the actual situation, resulting from the change of the road capacity under the conditions of rain, snow or fog, etc. The good effect can be seen in the example of this research as it effectively reduced delay by 19 % on average

Synthesis of a Triazaisotruxene-Based Porous Organic Polymer and Its Application in Iodine Capture

A new triazaisotruxene-based porous organic polymer (POP) was designed and successfully synthesized by a FeCl3-promoted crosslinking reaction. As a result of its porosity and good thermal stability, the designed POP can be utilized as a promising adsorbent for iodine, not only in the gaseous phase, but also in organic and aqueous solutions. Compared to its triazatruxene (TN) analogue, the ITN-based POP shows equal iodine uptake in the gaseous phase and in hexane solution, and better uptake in aqueous solution